TARGET AML Consensus Machine Learning: Supplemental Material
Sean Maden
Jenny Smith
This notebook contains supplemental findings for the manuscript "".
Cluster Analysis
Principle component analysis (PCA) and t-SNE (TSNE) clustering for dimensionality reduction was performed. From PCA, Feature-specific variance contributions to either PC1 or PCs 1, 2, and 3 were obtained and correlated with feature importance metrics from model fitting.
Figure 1. PCA analysis summary. (top) component-specific variances explained among top components. (bottom) projection of dataset on first 2 components, with coloration on (left to right) risk group, binary risk group, and primary cytogenetic group, respectively.
Figure 2. Tuning t-SNE perplexity metrics, with coloration on binary risk group (top) and primary cytogenetic category (bottom) for each rep.
Feature Importance Correlation Analysis
To reconcile findings from multiple algorithms, we initially performed correlation tests of the rank of feature importance as defined for various settings. Table S1 summarizes the importance metrics and conditions for each test type analyzed. We perfromed pairwise correlations of the actual importance measures, and, in the cases of lasso and svm, both actual and absolute values (Suppl. Fig. 3).
kable(toi, caption="Supplemental Table 1. Ontology terms of interest, and their free text labels, targeted for manual validation.") %>%
kable_styling(font_size = 12)
Figure 3. Correlation of feature importnace ranks across modeling algorithms and strategies. A-C: Correlation of actual importance ranks. D-F: Correlations of absolute importance ranks (Lasso and SVM absolute importances).
Consensus Important Features
We identified the most important feature subset identified in each strategy (Table 1, ‘imp_criteria’ column). This resulted in selection of N = 937 important features. Figure 4 shows an upset plot of the extent of consensus in important features selected using the various strategies and algorithms.
Figure 4. Upset plot of important feature membership across modeling algorithms and strategies.
Gene Set Enrichment Tests
LS0tCnRpdGxlOiAiVEFSR0VUIEFNTCBDb25zZW5zdXMgTWFjaGluZSBMZWFybmluZzogU3VwcGxlbWVudGFsIE1hdGVyaWFsIgpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sKYXV0aG9yOiAKLSBTZWFuIE1hZGVuCi0gSmVubnkgU21pdGgKLS0tCgpUaGlzIG5vdGVib29rIGNvbnRhaW5zIHN1cHBsZW1lbnRhbCBmaW5kaW5ncyBmb3IgdGhlIG1hbnVzY3JpcHQgIiIuIAoKYGBge3Igb3B0cywgZWNobz1GLCBldmFsPVR9CmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvPUYsIGV2YWw9RikKYGBgCgpgYGB7ciwgZGVwZW5kZW5jaWVzLCBlY2hvPUYsIGV2YWw9Rn0KbGlicmFyeShTdW1tYXJpemVkRXhwZXJpbWVudCkKbGlicmFyeShncmlkRXh0cmEpCmxpYnJhcnkoZ2dwbG90MikKbGlicmFyeShGYWN0b01pbmVSKQpsaWJyYXJ5KGZhY3RvZXh0cmEpCmxpYnJhcnkoZ2diaXBsb3QpCmxpYnJhcnkoZW5yaWNoUikKbGlicmFyeShVcFNldFIpCmxpYnJhcnkoZ2djb3JycGxvdCkKbGlicmFyeShSdHNuZSkKbGlicmFyeShrbml0cikKbGlicmFyeShrYWJsZUV4dHJhKQoKbG9hZCgiZGF0YS9zZXNldGZpbHRfZGVnc2VhaGFja190YXJnZXRhbWwucmRhIikKZGF0YSA9IGFzc2F5KGRlZ2ZpbHQuc2UpCnN0IDwtIHJlYWQuY3N2KCJkYXRhL3N0YW5kb3V0dGFibGUuY3N2IikKCmBgYAoKIyBDbHVzdGVyIEFuYWx5c2lzCgpQcmluY2lwbGUgY29tcG9uZW50IGFuYWx5c2lzIChQQ0EpIGFuZCB0LVNORSAoVFNORSkgY2x1c3RlcmluZyBmb3IgZGltZW5zaW9uYWxpdHkgcmVkdWN0aW9uIHdhcyBwZXJmb3JtZWQuIEZyb20gUENBLCBGZWF0dXJlLXNwZWNpZmljIHZhcmlhbmNlIGNvbnRyaWJ1dGlvbnMgdG8gZWl0aGVyIFBDMSBvciBQQ3MgMSwgMiwgYW5kIDMgd2VyZSBvYnRhaW5lZCBhbmQgY29ycmVsYXRlZCB3aXRoIGZlYXR1cmUgaW1wb3J0YW5jZSBtZXRyaWNzIGZyb20gbW9kZWwgZml0dGluZy4KCmBgYHtyLCBjbHVzdGVyYW5hbHlzaXMsIGVjaG89RiwgZXZhbD1GfQojIHJ1biBjbHVzdGVyIGFuYWx5c2VzCmRhdGFfcGNhIDwtIHByY29tcCh0KGRhdGEpLCBzY2FsZS49VCkKcmVzLnBjYSA8LSBQQ0EodChkYXRhKSwgZ3JhcGggPSBGQUxTRSkKdHNuZSA8LSBSdHNuZSh0KGRhdGEpLCBwZXJwbGV4aXR5PTMwKQpgYGAKCmBgYHtyLCBwY2F2aXosIGV2YWw9RiwgZWNobz1GfQojIHZpc3VhbGl6ZSBwY2EgZmluZGluZ3MKcDEgPC0gZnZpel9zY3JlZXBsb3QocmVzLnBjYSwgbmNwPTIwKQpwMiA8LSBnZ2JpcGxvdChkYXRhX3BjYSwgZWxsaXBzZT1UUlVFLCBncm91cHM9ZGVnZmlsdC5zZSRSaXNrLmdyb3VwLCB2YXIuYXhlcz1GQUxTRSkgKyBnZ3RpdGxlKCJCaXBsb3QgKEdyb3VwID0gUmlzayBHcm91cCkiKQpwMyA8LSBnZ2JpcGxvdChkYXRhX3BjYSwgZWxsaXBzZT1UUlVFLCBncm91cHM9ZGVnZmlsdC5zZSRkZWcucmlzaywgdmFyLmF4ZXM9RkFMU0UpICsgZ2d0aXRsZSgiQmlwbG90IChHcm91cCA9IEJpbmFyeSBSaXNrIEdyb3VwKSIpCnA0IDwtIGdnYmlwbG90KGRhdGFfcGNhLCBlbGxpcHNlPVRSVUUsIGdyb3Vwcz1kZWdmaWx0LnNlJFByaW1hcnkuQ3l0b2dlbmV0aWMuQ29kZSwgdmFyLmF4ZXM9RkFMU0UpICsgZ2d0aXRsZSgiQmlwbG90IChHcm91cCA9IEN5dG9nZW5ldGljIEdyb3VwKSIpCgpqcGVnKCJwY2FzdW1tYXJ5X211bHRpcGxvdC5qcGciLCAxMiwgMTAsIHVuaXRzPSJpbiIsIHJlcz00MDApCmdyaWQuYXJyYW5nZShwMSxwMixwMyxwNCxsYXlvdXRfbWF0cml4PW1hdHJpeChjKDEsMSwxLDIsMyw0KSwgbnJvdz0yLCBieXJvdz1UUlVFKSkKZGV2Lm9mZigpCmBgYAoKYGBge3IsIHRzbmV2aXosIGV2YWw9RiwgZWNobz1GfQojIHZpc3VhbGl6ZSB0c25lIGZpbmRpbmdzCmNvbGdycDEgPC0gaWZlbHNlKGRlZ2ZpbHQuc2UkZGVnLnJpc2s9PTAsICJyZWQiLCAiYmx1ZSIpCnYyIDwtIGRlZ2ZpbHQuc2UkUHJpbWFyeS5DeXRvZ2VuZXRpYy5Db2RlCmNvbGdycDIgPC0gaWZlbHNlKHYyPT0iaW52KDE2KSIsICJyZWQiLAogICAgICAgICAgICAgICAgICBpZmVsc2UodjI9PSJNTEwiLCJ5ZWxsb3ciLAogICAgICAgICAgICAgICAgICAgICAgICAgaWZlbHNlKHYyPT0iTm9ybWFsIiwiZ3JlZW4iLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGlmZWxzZSh2Mj09Ik90aGVyIiwiY3lhbiIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGlmZWxzZSh2Mj09InQoODsyMSkiLCAiYmx1ZSIsICJwdXJwbGUiKSkpKSkKCmpwZWcoInRzbmVwZXJwbGV4aXRpZXNfbXVsdGlwbG90LmpwZyIsIDE1LCA3LCB1bml0cz0iaW4iLCByZXM9NDAwKQoKcGFyKG1mY29sPWMoMiw2KSwgb21hPWMoNSw1LDQsMSkpCnB2bCA8LSBjKDEwLDIwLDMwLDQwLDQ1KQojIG1ha2UgcGxvdHMgdmFyeWluZyBwZXJwbGV4aXR5CnNldC5zZWVkKDIwMTkpCmZvcih0IGluIDE6NSl7CiAgcHR0eSA8LSBSdHNuZSh0KGRhdGEpLCBwZXJwbGV4aXR5PXB2bFt0XSkkWQogIHBsb3QocHR0eSwgCiAgICAgICBjb2w9Y29sZ3JwMSwgCiAgICAgICBtYWluPXBhc3RlMCgiUCA9ICIscHZsW3RdKSwgeGxhYj0iIiwgeWxhYj0iIikKICBwbG90KHB0dHksIAogICAgICAgY29sPWNvbGdycDIsIG1haW49IiIsIHhsYWI9IiIsIHlsYWI9IiIpCn0KIyByaXNrIGdyb3VwIGxlZ2VuZApwbG90Lm5ldygpCmxlZ2VuZCgidG9wbGVmdCIsIAogICAgICAgbGVnZW5kPWMoIjAiLCIxIiksIAogICAgICAgY29sPWMoInJlZCIsImJsdWUiKSwgCiAgICAgICBwY2g9YygxNiwxNiksIHRpdGxlPSJCaW5hcnkgUmlzayBHcm91cCIpCiMgY3l0b2dlbmV0aWMgZ3JvdXAgbGVnZW5kCnBsb3QubmV3KCkKbGVnZW5kKCJsZWZ0IiwgCiAgICAgICBsZWdlbmQ9YygiaW52KDE2KSIsICJNTEwiLCAiTm9ybWFsIiwgIk90aGVyIiwgInQoODsyMSkiLCAiVW5rbm93biIpLCAKICAgICAgIGNvbD1jb2xncnAyLCAKICAgICAgIHBjaD1jKHJlcCgxNiwgNikpLCB0aXRsZT0iUHJpbWFyeSBDeXRvZ2VuZXRpYyBHcm91cCIpCm10ZXh0KCJUU05FX0F4aXMxIiwgc2lkZT0xLGxpbmU9MixjZXg9MS4yLCBvdXRlcj1UKQptdGV4dCgiVFNORV9BeGlzMiIsIHNpZGU9MixsaW5lPTIsY2V4PTEuMiwgb3V0ZXI9VCkKbXRleHQoIlR1bmluZyBUU05FIFBlcnBsZXhpdGllcyIsIHNpZGU9MyxsaW5lPTEuOCxjZXg9Miwgb3V0ZXI9VCkKCmRldi5vZmYoKQpgYGAKCiFbRmlndXJlIDEuIFBDQSBhbmFseXNpcyBzdW1tYXJ5LiAodG9wKSBjb21wb25lbnQtc3BlY2lmaWMgdmFyaWFuY2VzIGV4cGxhaW5lZCBhbW9uZyB0b3AgY29tcG9uZW50cy4gKGJvdHRvbSkgcHJvamVjdGlvbiBvZiBkYXRhc2V0IG9uIGZpcnN0IDIgY29tcG9uZW50cywgd2l0aCBjb2xvcmF0aW9uIG9uIChsZWZ0IHRvIHJpZ2h0KSByaXNrIGdyb3VwLCBiaW5hcnkgcmlzayBncm91cCwgYW5kIHByaW1hcnkgY3l0b2dlbmV0aWMgZ3JvdXAsIHJlc3BlY3RpdmVseS5dKHBjYXN1bW1hcnlfbXVsdGlwbG90LmpwZyl7IHdpZHRoPTEwMCUgfQoKIVtGaWd1cmUgMi4gVHVuaW5nIHQtU05FIHBlcnBsZXhpdHkgbWV0cmljcywgd2l0aCBjb2xvcmF0aW9uIG9uIGJpbmFyeSByaXNrIGdyb3VwICh0b3ApIGFuZCBwcmltYXJ5IGN5dG9nZW5ldGljIGNhdGVnb3J5IChib3R0b20pIGZvciBlYWNoIHJlcC5dKHRzbmVwZXJwbGV4aXRpZXNfbXVsdGlwbG90LmpwZyl7IHdpZHRoPTEwMCUgfQoKIyBGZWF0dXJlIEltcG9ydGFuY2UgQ29ycmVsYXRpb24gQW5hbHlzaXMKClRvIHJlY29uY2lsZSBmaW5kaW5ncyBmcm9tIG11bHRpcGxlIGFsZ29yaXRobXMsIHdlIGluaXRpYWxseSBwZXJmb3JtZWQgY29ycmVsYXRpb24gdGVzdHMgb2YgdGhlIHJhbmsgb2YgZmVhdHVyZSBpbXBvcnRhbmNlIGFzIGRlZmluZWQgZm9yIHZhcmlvdXMgc2V0dGluZ3MuIFRhYmxlIFMxIHN1bW1hcml6ZXMgdGhlIGltcG9ydGFuY2UgbWV0cmljcyBhbmQgY29uZGl0aW9ucyBmb3IgZWFjaCB0ZXN0IHR5cGUgYW5hbHl6ZWQuIFdlIHBlcmZyb21lZCBwYWlyd2lzZSBjb3JyZWxhdGlvbnMgb2YgdGhlIGFjdHVhbCBpbXBvcnRhbmNlIG1lYXN1cmVzLCBhbmQsIGluIHRoZSBjYXNlcyBvZiBsYXNzbyBhbmQgc3ZtLCBib3RoIGFjdHVhbCBhbmQgYWJzb2x1dGUgdmFsdWVzIChTdXBwbC4gRmlnLiAzKS4KCmBgYHtyLCBkZWZ0YWJsZTEsIGV2YWw9VCwgZWNobz1GfQp0b2kgPC0gZGF0YS5mcmFtZSgic3RyYXRlZ3kiPWMoInBjYTEiLCJwY2ExMjMiLCJsYXNzbyAocmVwMSkiLCJzdm0gKHJlcDEpIiwicmYgKHJlcDQpIiwieGdib29zdCAocmVwNSkiKSwKICAgICAgICAgICAgICAgICAgImltcF9tZXRyaWMiPWMoInZhcmlhbmNlIGNvbnRyaWIuIiwgInZhcmlhbmNlIGNvbnRyaWIuIiwgIm1vZGVsIGNvZWZmaWNpZW50IiwgIm1vZGVsIHdlaWdodCIsICJpbXBvcnRhbmNlLCBtZWFuIGRlYy4gbm9kZSBwdXJpdHkiLCAiaW1wb3J0YW5jZSIpLAogICAgICAgICAgICAgICAgICAiaW1wX2NyaXRlcmlhIj1jKCI+bnVsbCIsICI+bnVsbCIsICJzZWxlY3Rpb24iLCAiPjAuMDA4ICg+OTB0aCBxdWFudGlsZSkiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAiPjAuMSBpbXBvcnRhbmNlICg+OTB0aCBxdWFudGlsZSkiLCJzZWxlY3Rpb24iKSkKYGBgCgpgYGB7ciwgbWFrZXRhYmxlMSwgZXZhbD1ULCBlY2hvPVQsIG1lc3NhZ2U9VCwgZmlnLmtlZXA9ImFsbCJ9CmthYmxlKHRvaSwgY2FwdGlvbj0iU3VwcGxlbWVudGFsIFRhYmxlIDEuIE9udG9sb2d5IHRlcm1zIG9mIGludGVyZXN0LCBhbmQgdGhlaXIgZnJlZSB0ZXh0IGxhYmVscywgdGFyZ2V0ZWQgZm9yIG1hbnVhbCB2YWxpZGF0aW9uLiIpICU+JQogIGthYmxlX3N0eWxpbmcoZm9udF9zaXplID0gMTIpCmBgYAoKYGBge3IsIGNvcnRlc3RzLCBldmFsPUYsIGVjaG89Rn0KIyByYW5rIGNvcnJlbGF0aW9ucwojIHVzZSBhY3R1YWwgdmFsdWVzIGZvciByYW5rcwpsYXNzbzFyYW5rID0gcmFuayhzdCRsYXNzb19jb2VmX3JlcDEpCnN2bTFyYW5rID0gcmFuayhzdCRzdm0xX3dlaWdodHMpCnhnNXJhbmsgPSByYW5rKHN0JHhnNV9pbXApCnJmMTBrcmFuayA9IHJhbmsoc3QkcmZuYl8xMGtfTWVhbkRlY05vZGVJbXApCiMgYXBwZW5kIHBjYSByYW5rcwpwY2RmIDwtIHBjY29udApwY2RmIDwtIHBjZGZbb3JkZXIobWF0Y2gocGNkZiRuYW1lLCBzdCRYKSksXQppZGVudGljYWwocGNkZiRuYW1lLCBzdCRYKQpwY3ZjMSA8LSByYW5rKHBjZGYkY29udF9wY2ExKQpwY3ZjMTIzIDwtIHJhbmsocGNkZiRjb250X3BjYTEyMykKIyByYW5rZGYKcmFua2RmLmFsbCA9IGRhdGEuZnJhbWUobGFzc28xPWxhc3NvMXJhbmssCiAgICAgICAgICAgICAgICAgICAgIHN2bTE9c3ZtMXJhbmssCiAgICAgICAgICAgICAgICAgICAgIHhnNT14ZzVyYW5rLAogICAgICAgICAgICAgICAgICAgICByZjEwaz1yZjEwa3JhbmssCiAgICAgICAgICAgICAgICAgICAgIHBjYzE9cGN2YzEsCiAgICAgICAgICAgICAgICAgICAgIHBjYzEyMz1wY3ZjMTIzKQoKY29ybWF0LmFsbCA8LSByb3VuZChjb3IocmFua2RmLmFsbCxtZXRob2Q9InNwZWFybWFuIiksMykKCiMgY29ycmVsYXRpb24gYW5hbHlzaXMgYW1vbmcgZ2VuZSBzdWJzZXRzCiMgYW1vbmcgbGFzc28xIGdlbmVzLCB3aXRoIHJlcmFua2luZwpzdHN1YiA8LSBzdFshc3QkbGFzc29fY29lZl9yZXAxPT0wLF0KcGNkZnN1YiA8LSBwY2RmWyFzdCRsYXNzb19jb2VmX3JlcDE9PTAsXQoKbGFzc28xcmFuayA9IHJhbmsoc3RzdWIkbGFzc29fY29lZl9yZXAxKQpzdm0xcmFuayA9IHJhbmsoc3RzdWIkc3ZtMV93ZWlnaHRzKQp4ZzVyYW5rID0gcmFuayhzdHN1YiR4ZzVfaW1wKQpyZjEwa3JhbmsgPSByYW5rKHN0c3ViJHJmbmJfMTBrX01lYW5EZWNOb2RlSW1wKQpwY3ZjMXN1YiA8LSByYW5rKHBjZGZzdWIkY29udF9wY2ExKQpwY3ZjMTIzc3ViIDwtIHJhbmsocGNkZnN1YiRjb250X3BjYTEyMykKCnJhbmtkZi5zdWIgPSBkYXRhLmZyYW1lKGxhc3NvMT1sYXNzbzFyYW5rLAogICAgICAgICAgICAgICAgICAgICAgICBzdm0xPXN2bTFyYW5rLAogICAgICAgICAgICAgICAgICAgICAgICB4ZzU9eGc1cmFuaywKICAgICAgICAgICAgICAgICAgICAgICAgcmYxMGs9cmYxMGtyYW5rLAogICAgICAgICAgICAgICAgICAgICAgICBwY2MxPXBjdmMxc3ViLAogICAgICAgICAgICAgICAgICAgICAgICBwY2MxMjM9cGN2YzEyM3N1YikKCmNvcm1hdC5sYXNzbzEgPC0gcm91bmQoY29yKHJhbmtkZi5zdWIsIG1ldGhvZD0ic3BlYXJtYW4iKSwzKQoKIyBhbW9uZyB4ZzUgZ2VuZXMsIHdpdGggcmVyYW5raW5nCnN0c3ViIDwtIHN0WyFzdCR4ZzVfaW1wPT0wLF0KcGNkZnN1YiA8LSBwY2RmWyFzdCR4ZzVfaW1wPT0wLF0KCmxhc3NvMXJhbmsgPSByYW5rKHN0c3ViJGxhc3NvX2NvZWZfcmVwMSkKc3ZtMXJhbmsgPSByYW5rKHN0c3ViJHN2bTFfd2VpZ2h0cykKeGc1cmFuayA9IHJhbmsoc3RzdWIkeGc1X2ltcCkKcmYxMGtyYW5rID0gcmFuayhzdHN1YiRyZm5iXzEwa19NZWFuRGVjTm9kZUltcCkKcGN2YzFzdWIgPC0gcmFuayhwY2Rmc3ViJGNvbnRfcGNhMSkKcGN2YzEyM3N1YiA8LSByYW5rKHBjZGZzdWIkY29udF9wY2ExMjMpCgpyYW5rZGYuc3ViID0gZGF0YS5mcmFtZShsYXNzbzE9bGFzc28xcmFuaywKICAgICAgICAgICAgICAgICAgICAgICAgc3ZtMT1zdm0xcmFuaywKICAgICAgICAgICAgICAgICAgICAgICAgeGc1PXhnNXJhbmssCiAgICAgICAgICAgICAgICAgICAgICAgIHJmMTBrPXJmMTBrcmFuaywKICAgICAgICAgICAgICAgICAgICAgICAgcGNjMT1wY3ZjMXN1YiwKICAgICAgICAgICAgICAgICAgICAgICAgcGNjMTIzPXBjdmMxMjNzdWIpCgpjb3JtYXQueGc1Z2VuZXMgPC0gcm91bmQoY29yKHJhbmtkZi5zdWIsIG1ldGhvZD0ic3BlYXJtYW4iKSwzKQoKIyBjb3JyIHdpdGggQUJTT0xVVEUgaW1wb3J0YW5jZSByYW5rcwpsYXNzbzFyYW5rLmFicyA9IHJhbmsoYWJzKHN0JGxhc3NvX2NvZWZfcmVwMSkpCnN2bTFyYW5rLmFicyA9IHJhbmsoYWJzKHN0JHN2bTFfd2VpZ2h0cykpCnhnNXJhbmsgPSByYW5rKHN0JHhnNV9pbXApCnJmMTBrcmFuayA9IHJhbmsoc3QkcmZuYl8xMGtfTWVhbkRlY05vZGVJbXApCiMgYXBwZW5kIHBjYSByYW5rcwpwY2RmIDwtIHBjY29udApwY2RmIDwtIHBjZGZbb3JkZXIobWF0Y2gocGNkZiRuYW1lLCBzdCRYKSksXQppZGVudGljYWwocGNkZiRuYW1lLCBzdCRYKQpwY3ZjMSA8LSByYW5rKHBjZGYkY29udF9wY2ExKQpwY3ZjMTIzIDwtIHJhbmsocGNkZiRjb250X3BjYTEyMykKIyByYW5rZGYKcmFua2RmLmFsbC5hYnMgPSBkYXRhLmZyYW1lKGxhc3NvMXJhbmsuYWJzPWxhc3NvMXJhbmsuYWJzLAogICAgICAgICAgICAgICAgICAgICAgICBzdm0xcmFuay5hYnM9c3ZtMXJhbmsuYWJzLAogICAgICAgICAgICAgICAgICAgICAgICB4ZzU9eGc1cmFuaywKICAgICAgICAgICAgICAgICAgICAgICAgcmYxMGs9cmYxMGtyYW5rLAogICAgICAgICAgICAgICAgICAgICAgICBwY2MxPXBjdmMxLAogICAgICAgICAgICAgICAgICAgICAgICBwY2MxMjM9cGN2YzEyMykKCmNvcm1hdC5hbGwuYWJzIDwtIHJvdW5kKGNvcihyYW5rZGYuYWxsLmFicywgbWV0aG9kPSJzcGVhcm1hbiIpLDMpCgojIGFtb25nIGxhc3NvMSBnZW5lcywgd2l0aCByZXJhbmtpbmcKc3RzdWIgPC0gc3RbIXN0JGxhc3NvX2NvZWZfcmVwMT09MCxdCnBjZGZzdWIgPC0gcGNkZlshc3QkbGFzc29fY29lZl9yZXAxPT0wLF0KCmxhc3NvMXJhbmsuYWJzID0gcmFuayhhYnMoc3RzdWIkbGFzc29fY29lZl9yZXAxKSkKc3ZtMXJhbmsuYWJzID0gcmFuayhhYnMoc3RzdWIkc3ZtMV93ZWlnaHRzKSkKeGc1cmFuayA9IHJhbmsoc3RzdWIkeGc1X2ltcCkKcmYxMGtyYW5rID0gcmFuayhzdHN1YiRyZm5iXzEwa19NZWFuRGVjTm9kZUltcCkKcGN2YzFzdWIgPC0gcmFuayhwY2Rmc3ViJGNvbnRfcGNhMSkKcGN2YzEyM3N1YiA8LSByYW5rKHBjZGZzdWIkY29udF9wY2ExMjMpCgpyYW5rZGYuc3ViLmFicyA9IGRhdGEuZnJhbWUobGFzc28xLmFicz1sYXNzbzFyYW5rLmFicywKICAgICAgICAgICAgICAgICAgICAgICAgc3ZtMS5hYnM9c3ZtMXJhbmsuYWJzLAogICAgICAgICAgICAgICAgICAgICAgICB4ZzU9eGc1cmFuaywKICAgICAgICAgICAgICAgICAgICAgICAgcmYxMGs9cmYxMGtyYW5rLAogICAgICAgICAgICAgICAgICAgICAgICBwY2MxPXBjdmMxc3ViLAogICAgICAgICAgICAgICAgICAgICAgICBwY2MxMjM9cGN2YzEyM3N1YikKCmNvcm1hdC5sYXNzbzEuYWJzIDwtIHJvdW5kKGNvcihyYW5rZGYuc3ViLmFicywgbWV0aG9kPSJzcGVhcm1hbiIpLDMpCgojIGFtb25nIHhnNSBnZW5lcywgd2l0aCByZXJhbmtpbmcKc3RzdWIgPC0gc3RbIXN0JHhnNV9pbXA9PTAsXQpwY2Rmc3ViIDwtIHBjZGZbIXN0JHhnNV9pbXA9PTAsXQoKbGFzc28xcmFuay5hYnMgPSByYW5rKGFicyhzdHN1YiRsYXNzb19jb2VmX3JlcDEpKQpzdm0xcmFuay5hYnMgPSByYW5rKGFicyhzdHN1YiRzdm0xX3dlaWdodHMpKQp4ZzVyYW5rID0gcmFuayhzdHN1YiR4ZzVfaW1wKQpyZjEwa3JhbmsgPSByYW5rKHN0c3ViJHJmbmJfMTBrX01lYW5EZWNOb2RlSW1wKQpwY3ZjMXN1YiA8LSByYW5rKHBjZGZzdWIkY29udF9wY2ExKQpwY3ZjMTIzc3ViIDwtIHJhbmsocGNkZnN1YiRjb250X3BjYTEyMykKCnJhbmtkZi5zdWIuYWJzID0gZGF0YS5mcmFtZShsYXNzbzEuYWJzPWxhc3NvMXJhbmsuYWJzLAogICAgICAgICAgICAgICAgICAgICAgICBzdm0xLmFicz1zdm0xcmFuay5hYnMsCiAgICAgICAgICAgICAgICAgICAgICAgIHhnNT14ZzVyYW5rLAogICAgICAgICAgICAgICAgICAgICAgICByZjEwaz1yZjEwa3JhbmssCiAgICAgICAgICAgICAgICAgICAgICAgIHBjYzE9cGN2YzFzdWIsCiAgICAgICAgICAgICAgICAgICAgICAgIHBjYzEyMz1wY3ZjMTIzc3ViKQoKY29ybWF0LnhnNWdlbmVzLmFicyA8LSByb3VuZChjb3IocmFua2RmLnN1Yi5hYnMsIG1ldGhvZD0ic3BlYXJtYW4iKSwzKQoKYGBgCgpgYGB7ciwgY29ydml6LCBlY2hvPUYsIGV2YWw9Rn0KY3AxIDwtIGdnY29ycnBsb3QoY29ybWF0LmFsbCwgaGMub3JkZXIgPSBGQUxTRSwgdHlwZSA9ICJsb3dlciIsCiAgICAgICAgICAgICAgICAgIG91dGxpbmUuY29sID0gIndoaXRlIiwKICAgICAgICAgICAgICAgICAgbGVnZW5kLnRpdGxlPSJSaG8iLCB0aXRsZT0iQWxsIERFR3MiKSArCiAgZ2VvbV90aWxlKGNvbG91ciA9ICJibGFjayIpICsKICBnZW9tX3RleHQoYWVzKGxhYmVsID0gcm91bmQodmFsdWUsIDEpKSkKCmNwMiA8LSBnZ2NvcnJwbG90KGNvcm1hdC5sYXNzbzEsIGhjLm9yZGVyID0gRkFMU0UsIHR5cGUgPSAibG93ZXIiLAogICAgICAgICAgICAgICAgICBvdXRsaW5lLmNvbCA9ICJ3aGl0ZSIsCiAgICAgICAgICAgICAgICAgIGxlZ2VuZC50aXRsZT0iUmhvIiwgdGl0bGU9Ikxhc3NvMSBHZW5lcyIpICsKICBnZW9tX3RpbGUoY29sb3VyID0gImJsYWNrIikgKwogIGdlb21fdGV4dChhZXMobGFiZWwgPSByb3VuZCh2YWx1ZSwgMSkpKQoKY3AzIDwtIGdnY29ycnBsb3QoY29ybWF0LnhnNWdlbmVzLCBoYy5vcmRlciA9IEZBTFNFLCB0eXBlID0gImxvd2VyIiwKICAgICAgICAgICAgICAgICAgb3V0bGluZS5jb2wgPSAid2hpdGUiLAogICAgICAgICAgICAgICAgICBsZWdlbmQudGl0bGU9IlJobyIsIHRpdGxlPSJYR0I1IEdlbmVzIikgKwogIGdlb21fdGlsZShjb2xvdXIgPSAiYmxhY2siKSArCiAgZ2VvbV90ZXh0KGFlcyhsYWJlbCA9IHJvdW5kKHZhbHVlLCAxKSkpCgpjcDQgPC0gZ2djb3JycGxvdChjb3JtYXQuYWxsLmFicywgaGMub3JkZXIgPSBGQUxTRSwgdHlwZSA9ICJsb3dlciIsCiAgICAgICAgICAgICAgICAgIG91dGxpbmUuY29sID0gIndoaXRlIiwKICAgICAgICAgICAgICAgICAgbGVnZW5kLnRpdGxlPSJSaG8gKEFicy4pIiwgdGl0bGU9IkFsbCBERUdzIikgKwogIGdlb21fdGlsZShjb2xvdXIgPSAiYmxhY2siKSArCiAgZ2VvbV90ZXh0KGFlcyhsYWJlbCA9IHJvdW5kKHZhbHVlLCAxKSkpCgpjcDUgPC0gZ2djb3JycGxvdChjb3JtYXQubGFzc28xLmFicywgaGMub3JkZXIgPSBGQUxTRSwgdHlwZSA9ICJsb3dlciIsCiAgICAgICAgICAgICAgICAgIG91dGxpbmUuY29sID0gIndoaXRlIiwKICAgICAgICAgICAgICAgICAgbGVnZW5kLnRpdGxlPSJSaG8gKEFicy4pIiwgdGl0bGU9Ikxhc3NvMSBHZW5lcyIpICsKICBnZW9tX3RpbGUoY29sb3VyID0gImJsYWNrIikgKwogIGdlb21fdGV4dChhZXMobGFiZWwgPSByb3VuZCh2YWx1ZSwgMSkpKQoKY3A2IDwtIGdnY29ycnBsb3QoY29ybWF0LnhnNWdlbmVzLmFicywgaGMub3JkZXIgPSBGQUxTRSwgdHlwZSA9ICJsb3dlciIsCiAgICAgICAgICAgICAgICAgIG91dGxpbmUuY29sID0gIndoaXRlIiwKICAgICAgICAgICAgICAgICAgbGVnZW5kLnRpdGxlPSJSaG8gKEFicy4pIiwgdGl0bGU9IlhHQjUgR2VuZXMiKSArCiAgZ2VvbV90aWxlKGNvbG91ciA9ICJibGFjayIpICsKICBnZW9tX3RleHQoYWVzKGxhYmVsID0gcm91bmQodmFsdWUsIDEpKSkKCiMgd2l0aCBsZXR0ZXJpbmcgaW5zZXQgbGFiZWxzCmNwMSA8LSBhcnJhbmdlR3JvYihjcDEsIAogICAgICAgICAgICAgICAgICAgdG9wID0gdGV4dEdyb2IoIkEuIiwgeCA9IHVuaXQoMCwgIm5wYyIpLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHkgICA9IHVuaXQoMSwgIm5wYyIpLCBqdXN0PWMoImxlZnQiLCJ0b3AiKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGdwPWdwYXIoY29sPSJibGFjayIsIGZvbnRzaXplPTE4LCBmb250ZmFtaWx5PSJBcmlhbCIpKSkKY3AyIDwtIGFycmFuZ2VHcm9iKGNwMiwgCiAgICAgICAgICAgICAgICAgICB0b3AgPSB0ZXh0R3JvYigiQi4iLCB4ID0gdW5pdCgwLCAibnBjIiksIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeSAgID0gdW5pdCgxLCAibnBjIiksIGp1c3Q9YygibGVmdCIsInRvcCIpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ3A9Z3Bhcihjb2w9ImJsYWNrIiwgZm9udHNpemU9MTgsIGZvbnRmYW1pbHk9IkFyaWFsIikpKQpjcDMgPC0gYXJyYW5nZUdyb2IoY3AzLCAKICAgICAgICAgICAgICAgICAgIHRvcCA9IHRleHRHcm9iKCJDLiIsIHggPSB1bml0KDAsICJucGMiKSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB5ICAgPSB1bml0KDEsICJucGMiKSwganVzdD1jKCJsZWZ0IiwidG9wIiksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBncD1ncGFyKGNvbD0iYmxhY2siLCBmb250c2l6ZT0xOCwgZm9udGZhbWlseT0iQXJpYWwiKSkpCmNwNCA8LSBhcnJhbmdlR3JvYihjcDQsIAogICAgICAgICAgICAgICAgICAgdG9wID0gdGV4dEdyb2IoIkQuIiwgeCA9IHVuaXQoMCwgIm5wYyIpLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHkgICA9IHVuaXQoMSwgIm5wYyIpLCBqdXN0PWMoImxlZnQiLCJ0b3AiKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGdwPWdwYXIoY29sPSJibGFjayIsIGZvbnRzaXplPTE4LCBmb250ZmFtaWx5PSJBcmlhbCIpKSkKY3A1IDwtIGFycmFuZ2VHcm9iKGNwNSwgCiAgICAgICAgICAgICAgICAgICB0b3AgPSB0ZXh0R3JvYigiRS4iLCB4ID0gdW5pdCgwLCAibnBjIiksIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeSAgID0gdW5pdCgxLCAibnBjIiksIGp1c3Q9YygibGVmdCIsInRvcCIpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ3A9Z3Bhcihjb2w9ImJsYWNrIiwgZm9udHNpemU9MTgsIGZvbnRmYW1pbHk9IkFyaWFsIikpKQpjcDYgPC0gYXJyYW5nZUdyb2IoY3A2LCAKICAgICAgICAgICAgICAgICAgIHRvcCA9IHRleHRHcm9iKCJGLiIsIHggPSB1bml0KDAsICJucGMiKSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB5ICAgPSB1bml0KDEsICJucGMiKSwganVzdD1jKCJsZWZ0IiwidG9wIiksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBncD1ncGFyKGNvbD0iYmxhY2siLCBmb250c2l6ZT0xOCwgZm9udGZhbWlseT0iQXJpYWwiKSkpCgpqcGVnKCJjb3JobW11bHRpXzZwbG90c19pbnNldGxhYnNfY29uc2Vuc3VzaW1wcmFuay5qcGciLCAxMiwgMTAsIHVuaXRzPSJpbiIsIHJlcz00MDApCmdyaWQuYXJyYW5nZShjcDEsIGNwMiwgY3AzLCBjcDQsIGNwNSwgY3A2LCBuY29sPTMpCmRldi5vZmYoKQoKYGBgCiFbRmlndXJlIDMuIENvcnJlbGF0aW9uIG9mIGZlYXR1cmUgaW1wb3J0bmFjZSByYW5rcyBhY3Jvc3MgbW9kZWxpbmcgYWxnb3JpdGhtcyBhbmQgc3RyYXRlZ2llcy4gQS1DOiBDb3JyZWxhdGlvbiBvZiBhY3R1YWwgaW1wb3J0YW5jZSByYW5rcy4gRC1GOiBDb3JyZWxhdGlvbnMgb2YgYWJzb2x1dGUgaW1wb3J0YW5jZSByYW5rcyAoTGFzc28gYW5kIFNWTSBhYnNvbHV0ZSBpbXBvcnRhbmNlcykuXShjb3JobW11bHRpXzZwbG90c19pbnNldGxhYnNfY29uc2Vuc3VzaW1wcmFuay5qcGcpeyB3aWR0aD0xMDAlIH0KCiMgQ29uc2Vuc3VzIEltcG9ydGFudCBGZWF0dXJlcwoKV2UgaWRlbnRpZmllZCB0aGUgbW9zdCBpbXBvcnRhbnQgZmVhdHVyZSBzdWJzZXQgaWRlbnRpZmllZCBpbiBlYWNoIHN0cmF0ZWd5IChUYWJsZSAxLCAnaW1wX2NyaXRlcmlhJyBjb2x1bW4pLiBUaGlzIHJlc3VsdGVkIGluIHNlbGVjdGlvbiBvZiBOID0gOTM3IGltcG9ydGFudCBmZWF0dXJlcy4gRmlndXJlIDQgc2hvd3MgYW4gdXBzZXQgcGxvdCBvZiB0aGUgZXh0ZW50IG9mIGNvbnNlbnN1cyBpbiBpbXBvcnRhbnQgZmVhdHVyZXMgc2VsZWN0ZWQgdXNpbmcgdGhlIHZhcmlvdXMgc3RyYXRlZ2llcyBhbmQgYWxnb3JpdGhtcy4KCmBgYHtyLCBjb25zZW5zdXNfZGVnX3Vwc2V0LCBlY2hvPUYsIGV2YWw9Rn0KdXNpZyA8LSBsaXN0KCkKdXNpZ1tbImxhc3NvMSJdXSA9IGFzLmNoYXJhY3RlcihzdFshc3QkbGFzc29fY29lZl9yZXAxPT0wLF0kWCkKdXNpZ1tbInN2bTEiXV0gPSBhcy5jaGFyYWN0ZXIoc3RbYWJzKHN0JHN2bTFfd2VpZ2h0cyk+PTAuMDA4LF0kWCkKdXNpZ1tbInJmMTAiXV0gPSBhcy5jaGFyYWN0ZXIoc3RbYWJzKHN0JHJmbmJfMTBrX01lYW5EZWNOb2RlSW1wKT49MC4xLF0kWCkKdXNpZ1tbInhnNSJdXSA9IGFzLmNoYXJhY3RlcihzdFshc3QkeGc1X2ltcD09MCxdJFgpCnVzaWdbWyJwYzEiXV0gPSBwY2NvbnRbcGNjb250JGFib3ZlX3RoZW9pbXBfcGMxPT0iWSIsXSRuYW1lCnVzaWdbWyJwYzEyMyJdXSA9IHBjY29udFtwY2NvbnQkYWJvdmVfdGhlb2ltcF9wYzE9PSJZIixdJG5hbWUKCmZvcihpIGluIDE6bGVuZ3RoKHVzaWcpKXsKICBuYW1lcyh1c2lnKVtpXSA8LSBwYXN0ZTAobmFtZXModXNpZylbaV0sIiAoIixsZW5ndGgodXNpZ1tbaV1dKSwiIGdlbmVzKSIsY29sbGFwc2U9IiIpCn0KanBlZygidXBzZXRfY21sbW9kZWxnZW5lcy5qcGciLDcsNSx1bml0cz0iaW4iLCByZXM9NDAwKQp1cHNldChmcm9tTGlzdCh1c2lnKSwgb3JkZXIuYnkgPSAiZnJlcSIsIG5zZXRzPWxlbmd0aCh1c2lnKSwKICAgICAgc2V0cy5iYXIuY29sb3IgPSAiYmx1ZSIsIG1haW4uYmFyLmNvbG9yID0gInJlZCIpCmdyaWQudGV4dCgiT3ZlcmxhcCBpbiBTZWxlY3RlZCBhbmQgRmlsdGVyZWRcbk1vZGVsIEdlbmVzIiwgeD0wLjY1LCB5PTAuOTUsIGdwPWdwYXIoZm9udHNpemU9OSkpCmRldi5vZmYoKQpgYGAKCiFbRmlndXJlIDQuIFVwc2V0IHBsb3Qgb2YgaW1wb3J0YW50IGZlYXR1cmUgbWVtYmVyc2hpcCBhY3Jvc3MgbW9kZWxpbmcgYWxnb3JpdGhtcyBhbmQgc3RyYXRlZ2llcy5dKHVwc2V0X2NtbG1vZGVsZ2VuZXMuanBnKXsgd2lkdGg9NzAlIH0KCiMgR2VuZSBTZXQgRW5yaWNobWVudCBUZXN0cwoKCgo=